Selective Listening by Synchronizing Speech With Lips

A speaker extraction algorithm seeks to extract the speech of a target from multi-talker mixture when given cue that represents speaker, such as pre-enrolled utterance, or an accompanying video track. Visual cues are particularly useful is not available. In this work, we don’t rely on speaker’s speech, but rather use face track cue, referred auxiliary reference, form attractor towards speaker. We advocate temporal synchronization between and its lip movements direct dominant audio-visual cue. Therefore, propose self-supervised pre-training strategy, exploit speech-lip for extraction, which allows us leverage abundant unlabeled in-domain data. transfer knowledge pre-trained model encoder network. show proposed network outperforms various competitive baselines in terms signal quality, perceptual intelligibility, achieving state-of-the-art performance.